Sign in with
Sign up | Sign in
Your question

Intel vt-d for home vm use

Last response: in Business Computing
Share
February 4, 2012 2:15:59 AM

I'm putting together a new system and one of the things I want to do on it is run a VM. When choosing CPUs I came across Intel's vpro technology. I'm a comp-sci student and last semester we looked at VMs, hardware virtualization, how VM device use is slowed because it has to go through the native OS and how hardware virtualization would help alleviate this. Naturally, vt-d looked like a good idea so I've been trying to build a system around vpro support that includes vt-d. This is easier said than done though! I've chosen the i5 2500 and so far (for lack of better alternatives that I've found) the Q67 chipset (haven't got to motherboards yet).

I think I'm getting ahead of myself though. Before I continue banging my head against the wall, trying to dig up all this vpro stuff and worrying about compatibility and which alternative is better, I would like to find out if it's even worth it, whether it will do what I want or not. I'm choosing this stuff to make my VM run closer to native performance. Does anyone have experience with vt-d? Is it worth all of this headache?

More about : intel home

a b å Intel
February 4, 2012 3:21:11 AM

Well since you are a comp sci I would assume you would at least like to try it out. It's not as difficult as you think, for hardware, you just need a capable mobo and cpu.

Here is my low priced suggestions: xeon E3-1230 and Intel DQ67SW. Although the 1230 doesn't have igpu so you may want to get the 1235. I suggest this over the i5 because it has HT.
http://www.newegg.com/Product/Product.aspx?Item=N82E168...
http://www.newegg.com/Product/Product.aspx?Item=N82E168...
February 5, 2012 3:20:15 AM

Thanks! I'll definitely look into those.

You've also answered another question I had, which was whether or not the devices had to be 'vt-d compatible'. I didn't think they did but I wanted to be sure. Based on how it seems to work it would make sense that anything can be used. It looks like the vt-d is implemented by giving the OSs different address mappings for the devices or something along those lines (I haven't read the whole page yet but the section after the introduction talks a bit about it).
http://software.intel.com/en-us/articles/intel-virtuali...

I have another question related to vpro-compatible motherboards, but I'll post that in the 'chipsets and bios' section of the forum.

Thanks again!
Related resources
February 6, 2012 1:15:18 AM

What other plans do you have for this system? I mean, are you planning to run the free ESXi Hypervisor 5.0 or something else on it? What is the budget you're looking at for this home system?

Recently I've ordered several HP ProLiant ML110 G7 servers to run a single virtual machine within the free ESXi (both Windows 7 Professional and Windows Server 2008 R2) and have had wonderful luck with them. Granted, in all of them I added an HP SmartArray P410/256 MB RAID controller to enable full support for RAID arrays in the ESXi, but it has worked very well, and the performance has been great. These servers are quite affordable when you look at all the features you get with them, and from what I can see they have all your virtualization features with the Sandy Bridge Xeon processors.

Your only limitation with the ML110 G7 server is the maximum memory is 16 GB, but that's still quite a bit to throw at a single VM and well within the range to use the free ESXi Hypervisor.
February 9, 2012 12:18:10 AM

That chipset and bios thread disappeared on me. It was closed when I marked best answer and now I can't find it. No big deal, I was just going to refer people back to this thread. I'll look into the ASRock boards, the workstation/server chipsets and their boards and compare them to the aforementioned Intel board. The Intel board seems great but it doesn't have many PCIe expansion slots which I thought may become an issue.



The plans for the system are to use it for homework (includes programming), internet and some media (blu-ray viewing, itunes/music library). It's replacing the family desktop (we've had it for five to ten years). No gaming (I prefer consoles) and not to make it a general use public server. It may be used as a home media server though. With regards to virtualization, I've only ever really used VMs to develop in Linux and to have a sandbox environment. Those are currently my plans for it.

I'm not familiar with the ESXi Hypervisor (I know it comes from VMWare but I've never experienced it nor know how it's different from the Player) but I'll be using freeware. I'm thinking to spend about a grand. What do you use your servers for choucove?



Thanks guys, looks like I have some researching to keep me busy a while.
a b å Intel
February 9, 2012 5:31:51 AM

When you select a best answer, they tend to close the threads so they don't get bumped. You can click your name and see all of your posts. http://www.tomshardware.com/forum/306308-30-does-intel-...

The asrock z68 extreme3 is near the same price as the q67. I just noticed xeons are on the support list too, that's rare for a company to do that on a desktop chipset. http://www.newegg.com/Product/Product.aspx?Item=N82E168... Q67 and c206 have amt, the others do not so you should look into that. http://www.tomshardware.com/reviews/vpro-amt-management... Other than that there isn't anything else affecting virtualization. This is the only c206 available. http://www.newegg.com/Product/Product.aspx?Item=N82E168... What more research do you need to do? We could help with any questions you may have.
February 9, 2012 12:53:14 PM

These ML110 G7 servers have been used for various duties, primarily used as a simple storage server for some small offices here in town. We've done a few different configurations, basically with the ML110 G7 with a 3.3 Ghz Xeon quad-core processor, 8 GB of DDR3 1333 ECC memory, dual redundant 460 Watt power supplies, upgraded to HP SmartArray P410/256 MB RAID controller card, and put in two Western Digital Caviar Black or RE4 2 TB SATA hard drives in RAID 1. On some of the servers we have installed Windows Server 2008 R2 Standard and run Hyper-V with another instance of the virtual Windows Server 2008 R2 Standard to act as a file server/domain controller, but if a domain controller isn't necessary (which in most cases with these offices its not) then we just put a USB flash drive in the server, install ESXi Hypervisor, and run a virtual machine of Windows 7 Professional to do file sharing duties and manage automatic backups of the network. This setup would also work well for FreeNAS.

The problem with using ESXi Hypervisor in your particular situation is you would not be able to use the actual server as a standard desktop computer. What I mean is, when you install ESXi and it boots up from the flash drive, there is no standard desktop OS that you can go into to access email, install programs, play media, etc. etc. It's just a very stripped down basic GUI interface to allow you to change the IP address of the physical host server and change the administrative password. Everything else must be done and utilized through a completely different computer.

So, instead it seems what would be best for you would be VMWare's Player as it installs as an application on top of an existing operating system. This means that you will need a little more computational power since you have the host operating system, such as Windows 7, running as well as any virtual machines.

If you're looking to do virtual machines on a regular desktop system for testing and learning purposes, then about the only motherboard with the features you're looking for is the ASUS P8B WS http://www.newegg.com/Product/Product.aspx?Item=N82E168... which supports ECC memory and Xeon processors. However, from what I can tell it does NOT support VT-d for some reason.

While the ML110 G7 itself offers the features you're looking for in virtualization, it would be a little difficult to convert to a desktop usage system perhaps. For instance, you would need to install a dedicated graphics card, and would have to go with a single slot graphics card as the only PCI-e x16 slot is at the very bottom of the case, and there is no additional power connectors for PCI-e devices. You would also need to purchase an add-in sound card to get audio. If you're wanting to play Blu-Ray you will also need to replace the included DVD drive, and then you still have to add your hard drives (which also require purchasing hard drive sleds for the drive cage.) So, in the end it would not be possible within your budget I don't believe.
February 9, 2012 5:16:14 PM

Ah, thank you for that update, K1114 that is very good to hear as I had been disappointed before to hear the motherboard did not support VT-d.
February 14, 2012 6:13:00 AM

I have P8B WS, just recently tried to install esxi 5.0 and received a compatibility error. It also seems esxi 4.1 doesn't support this motherboard either.. installed with errors along the way.. was able to get to esxi command console but i think there is some issues. (cpu is e3 1270
a b å Intel
February 14, 2012 3:46:54 PM

Bios version? os?
February 15, 2012 12:36:56 AM

k1114 said:
Bios version? os?


latest bios 0900, installed esxi on physical.

received an error "This host has VT support however VT has not been enabled".
Thing is, I did enable it in the bios.
a b å Intel
February 15, 2012 2:23:26 AM

I'm not familiar with physical. You said you tried to install it and now you are saying you did install it. Is the error when installing or when executing.? Is there somewhere in the os or options you missed?
February 15, 2012 3:27:13 AM

k1114 said:
I'm not familiar with physical. You said you tried to install it and now you are saying you did install it. Is the error when installing or when executing.? Is there somewhere in the os or options you missed?


what i meant by physical is, non virtual. im not sure if i have the right terminology.
the beginning of the installation is when i received that error, but it still installed all the way through. not sure if esxi is utilizing the vt-d of the processor, due to that error.
a b å Intel
February 15, 2012 4:00:11 AM

The term is host which still doesn't tell me what os. But it's not a compatibility issue and it does say you can use vt. The error says it is supported by the host so your bios, os, hardware, everything is good to go. But you need to enable it in esxi. I'm guessing you are just learning esxi.
February 15, 2012 4:58:29 AM

Also AMD have support for IOMMU and perhaps it easier to come by a decent configuration for a decent price from AMD than from Intel. Most 990FX motherboards support IOMMU and you basically can't go wrong with the CPU.

Also Xen supports passthrough via VT-d/IOMMU. If it works? You could check this youtube video for example:

Xen Dual VGA Passthrough

Another thing you may want to check is whether the hardware that you want to pass through supports function level reset (FLR). Nvidia does supply firmware patches for their (later generation) Geforce boards when contacting support (if they don't support it already). Another way to reset hardware is by using D3-D0 power states where you cut the Vcc power to the PCIe slot via ACPI. You may want to check whether Xen supports this type of reset before you try it out as I'm not sure. DirectPath supports it though.
February 15, 2012 6:12:06 AM

From what I can tell the Q chipset is not necessary. I'm using an Asrock Z68 with a i5-2500 (not K not K not K). The VT-d's devices are :
- AMD 5850
- 2nd onboard USB2.0
- onboard audio (has issues)
- Creative X-Fi PCIe (not usable with native drivers)
- RTL8169 NIC

Failed:
- Creative Audigy 2 PCI
- Hercules Fortission IV (PCI VIA-based sound card)
- Geforce GTX260

Now, the question is whether the failed attemtps would have been better with Q chipset (since I found out that only they have "VT-d yes" on intel website).

Host: debian with Xen 4.1.2
Guest: Windows 7 (XP seems to not work right)

Just for the record, the 5850 is a secondary card to guest. The primary is the emulated one (which show a "!" once the 5850 is active). Same setup does not work for GTX260.
As for the issues for the sound cards, I would relate them to IRQ issues, because there are some pops, like when a sample arrives too late to the sound card. This cannot be tested/verified with any other device, as none are so time-sensitive (this is about ms-response times).
a b å Intel
February 15, 2012 4:05:11 PM

It won't be better on q67, the intel site only states intel's own mobos not the mobos for other companies. As stated in his other thread which the link was posted for, the bios needs to have support for it. Either it supports it or it doesn't. The hardware (gpu, sound card, sata card, etc) also need to support vt, which is why all your hardware didn't work. Other people have no issues with the 260 so you may want to troubleshoot that.
February 15, 2012 5:27:00 PM

It seems that AMD has better support for virtualization than Intel at least on the desktop platform. Basically all current lines of AMD CPUs support AMD-v (which is the same as Intel VT-x) whereas many of Intel's low-budget line of CPU's don't have it. VT-x/AMD-v is useful but as I understand it, the thread is about VT-d, or IOMMU which is a more general term for this feature. AMD also calls it AMD-Vi.

In the AMD platform the support for IOMMU is dictated by the chipset and not the CPU as it is for Intel's VT-d. Apart from the chipset, the BIOS must have proper ACPI IVRS tables and support for it (the same goes for Intel's VT-d). There were some problems with it on 890FX motherboards but there should be no problems on 990FX motherboards (the 990X and 970 chipsets also support IOMMU but I don't know if it is always implemented in the BIOS). When that is settled all compatible AMD CPU's that I'm aware of will work with IOMMU and VT-x.

When it comes to peripheral I/O devices, there is no "special" virtualization feature that they need in order to work with PCI passthrough. What you need is to be able to reset them by software without resetting the entire system. This can mainly be done by function level reset (FLR) by which you can reset individual hardware components (or PCI devices, onboard components such as the NIC are also PCI devices) at the function level. The other way is to use the power save/management features in the ACPI where you can cut power to the PCI devices. A lot of hardware that are not specifically designed for virtualization or passthrough can be reset this way, at least in VMWare's VMDirectPath.

I don't know about Xen which is the virtualization software that comes to mind when one talks about advanced virtualization. In their VT-d HowTo page it explicitly says that only hardware with FLR is supported so it seems that Xen does not support hardware reset via ACPI, or d3d0 as it is also called (the power states D3 to D0 are covered in the Wikipedia page about ACPI). So this limitation is not in the hardware but in Xen, so the best thing to do is to point this out for the development team and hope they will implement it some day in the future.

There are other ways that can be used to reset but then you are moving into more dangerous territory. More information about these methods can be found here: http://www.vmware.com/files/pdf/techpaper/vsp_4_vmdirec...
Of course it would be desirable if peripheral manufacturers provided BIOS patches for their hardware that enabled FLR support and eventually supported it by default. After all it ain't no rocket science and it is the preferred way to initialize the hardware for PCI passthrough.

Then there is VGA passthrough which is an advanced form of PCI passthrough where you also need to get the VGA BIOS mappings sorted out in order for it to work. I don't know about ATI (or AMD) but Nvidia seems to be diligent to get it supported and they apparently supply firmware patches upon request for their Geforce cards. I've also read somewhere that their lines of Quadro and Tesla cards support FLR by default and are specifically designed to work with PCI/VGA passthrough.
a b å Intel
February 15, 2012 8:20:11 PM

I don't know of any modern intel cpu that doesn't have vt-x. But vt-d is another story. Amd does make it a bit simpler by just supporting passthrough on all of their cpus. Although in a sense it still is dictated by the cpu, it's just all of them support it.

This thread has derailed after the hijack, I'm not even sure if the op is still here. Good info being posted but I think we should leave the thread to answering questions/resolving issues and I don't see any.
February 16, 2012 2:14:24 AM

Yeah, I'm still here. Time's spread kind of thin is all. First off I'd like to thank everyone for their interest and input, it's generated good information.

I had considered AMD chips early in my research. I hadn't really looked too much into or compared the kind of support for the technologies between AMD and Intel. Reading around though, it seems that (for this generation of processors at least) Intel gives better overall performance.

I looked at a couple of Xeon E3s and a couple of the suggested motherboards. I'm leaning toward the E3 1235 with the ASUS P8B WS (looks like it has more PCIe 2.0 slots than the ASRock which, I believe, translates into being able to add more expansion cards). Looks like a winning combination but I'll put off ordering until I'm happy with a set of peripherals (got to make sure they fit on it). (Thanks to k1114 for the suggestion!)

To be clear on which virtualization tools can take advantage of vt-d (please correct me if I'm wrong), VMWare only offers support with ESXi, VirtualBox can't use this (yet, no estimated implementation date) and XEN implements it standard. Sounds like the winner is XEN (for my situation). I've never used it though, don't know what it's like. Apparently you can't take snapshots with it (a well-liked feature of VirtualBox), but there seems to be at least one (time-consuming) workaround for Linux guests
(http://searchservervirtualization.techtarget.com/tip/Cr...).

Back to peripheral vt-d support, when researching the hardware will compatible parts say 'FLR implemented' or something in their description? Also, is FLR only needed for things that plug into PCI/PCIe slots and SATA plugs or, for example, does an external hard drive connected via USB need it too? I would think things that plug directly into the board (let's call them primary connectors) would need FLR and things that connect to those primary connectors don't. If that's the case then the external drive wouldn't need FLR because the USB microcontroller is the one that needs to have it.
February 16, 2012 4:05:43 AM

FLR is in the PCI-SIG specification, USB and SATA is a different beast. IOMMU is for passing through PCI devices (even when you talk about PCIe or PCI-X you mean PCI also here as the software doesn't differentiate between PCI and PCIe).

When it comes to USB you could either pass through the entire USB controller with PCI passthrough or use USB passthrough via the hypervisor's (e.g. Xen or VirtualBox) software interface.

As for SATA, if you intend to run a ZFS storage pool on a virtual Solaris/OpenIndiana machine and/or expect the VM to use particularly heavy disk access then it is recommended that you PCI pass through the entire disk controller or host bus adapter, otherwise there are other layers (than IOMMU) through which you can pass through hard drive storage to the VMs.

Also KVM and libvirt have support for IOMMU and VirtualBox have added some IOMMU support on the Linux version. This functionality most likely comes from the KVM project. VMWare has only mentioned IOMMU support in their vSphere/ESXi products and VMDirectpath. I'm not aware of any other VMWare products that support IOMMU.

If you are looking for VGA passthrough, currently only Xen has support for it. The other hypervisors can only pass through regular single- and multifunction PCI devices.

A problem with most hardware today is that it generally doesn't say on the package or description whether it supports FLR or not. In Linux you can find out by the lspci -vv command; If the hardware has a "FLReset+" in the DevCap field then it supports FLR. If it has a "FLReset–" then it doesn't support FLR. Also, most hardware out there don't have FLR, it is actually pretty uncommon unfortunately. Perhaps matthew7 could provide some insights how he got around these hurdles. I'm impressed that he managed to pass through an AMD 5850.
February 23, 2012 1:18:37 AM

Thank you very much threehosts! I think with the addition of that last piece of information I have what I need to give this project of mine some momentum. I've got in mind a cpu, chipset and board that support vt-d, I know to look for peripherals that support FLR ("function level reset" for people reading this as a summary) and I know of a couple of VMs that support vt-d/IOMMU. Please let me know if I'm overlooking anything...

When I finally build the computer I'll post my setup for people who are curious. From there I'll probably have questions setting up the software, but that's a little far off right now and I don't want to get ahead of myself...
April 21, 2012 6:16:46 AM

Anto_di_Pretoro said:
Thank you very much threehosts! I think with the addition of that last piece of information I have what I need to give this project of mine some momentum. I've got in mind a cpu, chipset and board that support vt-d, I know to look for peripherals that support FLR ("function level reset" for people reading this as a summary) and I know of a couple of VMs that support vt-d/IOMMU. Please let me know if I'm overlooking anything...

When I finally build the computer I'll post my setup for people who are curious. From there I'll probably have questions setting up the software, but that's a little far off right now and I don't want to get ahead of myself...



What VM's do you have in mind that support vt-d(exept esxi-vsphere) ? Because the only one that I can find is parallel and even in that I can't get it to work!
June 7, 2012 12:12:38 AM

Once again, thank you all for all of your help! = ) I believe I'm overdue to wrap this thread up.

Unfortunately it seems that there are currently no type 2 hypervisors with vt-d support, only a couple of type 1 hypervisors. I didn't try any of the type 1s; I want to use a type 2. VirtualBox is the only (free) one for which I could see evidence that they were working on vt-d, but it looks like they have a long way to go.

To conclude, unless you're planning on having your new computer for a good few years (like I am) or plan on using a type 1 hypervisor (whose use of vt-d I can't comment on) I would suggest passing on vt-d and saving some headache. Hopefully type 2 support will come around by your next build. In the meantime, it doesn't look like we'll be getting anything soon.


P.S. I know I said I'd post my setup but I'm going to skip it. I'll post it if anyone asks about it.
June 9, 2012 3:57:29 AM

Anto_di_Pretoro said:
Once again, thank you all for all of your help! = ) I believe I'm overdue to wrap this thread up.

Unfortunately it seems that there are currently no type 2 hypervisors with vt-d support, only a couple of type 1 hypervisors. I didn't try any of the type 1s; I want to use a type 2. VirtualBox is the only (free) one for which I could see evidence that they were working on vt-d, but it looks like they have a long way to go.

To conclude, unless you're planning on having your new computer for a good few years (like I am) or plan on using a type 1 hypervisor (whose use of vt-d I can't comment on) I would suggest passing on vt-d and saving some headache. Hopefully type 2 support will come around by your next build. In the meantime, it doesn't look like we'll be getting anything soon.


P.S. I know I said I'd post my setup but I'm going to skip it. I'll post it if anyone asks about it.


I would be very interested to see what you came up with for your setup! After reading this entire thread I've learned a lot, and have very similar interests to yours, Anto_di_Pretoro.

John
July 7, 2012 8:02:53 PM

Case: Fractal Design Define R3 Black ATX Mid Tower Silent Computer Case 2X5.25 8X3.5INT USB3.0 No PS (Part#: FD-CA-DEF-R3-USB3-BL)

PSU: CORSAIR Professional Series Gold AX850 (CMPSU-850AX) 850W ATX12V v2.31 / EPS12V v2.92 80 PLUS GOLD Certified Full Modular Active PFC Power Supply

CPU: Intel Xeon E3-1235 3.20 GHz Processor - Socket H2 LGA-1155 - Quad (stock cooler)
MB: ASUS P8B WS LGA 1155 Intel C206 ATX Intel Xeon E3 Server/Workstation Motherboard
RAM: 2 x Patriot Gamer 2 Series 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model PGD316G1333ELK (i.e. 32 GB ram)

SSD: Crucial 128 GB Internal Solid State Drive - 2.5" - SATA/600
Hard drive: Western Digital Caviar Black WD1002FAEX 1 TB 3.5" Internal

Monitor: VGA CRT
Monitor Adapter: Link Depot DVI-AVGA-ADPT DVI To VGA Adapter

Optical drive: LG Black 12X Super Multi Blue with 3D Playback & M-DISC Support SATA WH12LS39 LightScribe Support - OEM

PCI Peripherals: Hauppauge WinTV-HVR-2250 Media Center Kit - PCI Express x1 - ATS

OS: Windows 7 Professional (a note about using bluetooth mice and keyboards: the MB firmware detects and uses mine right away but Windows, during installation, didn't find them which forced me to go buy corded ones. Now, once at the login screen it takes too long in my opinion to enable them. Just saying...)

I'm using VirtualBox for the VM because I want a type 2 hypervisor and there's evidence VB is working on VT-d support.

Note that the case comes with a slot adapter for a 3.5" floppy drive but the motherboard doesn't have an IDE connector so if you want a floppy drive, get the USB ones (unless you have a SATA floppy drive but I've never seen one. Haven't looked though...).

Note also that if you're in Canada (Ottawa, specifically) like me, you would be better off to invest in a VCR and a bunch of blank tapes instead of the t.v. tuner! The only way I can get channels is through a coaxial cable hooked up to Rogers. They don't have very many analog channels coming through and the ones they do aren't very interesting. As for the digital channels (there are many more of them), they are encrypted and need the Rogers box to decode them. So to use my tuner I have to connect the cable to the rogers box, and then connect the box to the tuner with the 'cable out' jack. The worst part is, though, that you can only record the channel that the box is set to! That means that you can't watch one show while recording another.

As for the blu-ray drive, don't get an OEM one like I did. Get the software (you need special software to play blu-rays)! I wouldn't mind paying for the software separately (more expensive to buy them separate) if I **knew** it worked, but none of the different vendors offer a trial for blu-ray playback (my mistake was assuming they would. Anyone remember the SCTV 'children's' skit about 'assumptions'?). There wouldn't be a problem except that I'm using a CRT. When it comes to CRTs, they MAY play the blu-rays and they might NOT. The reason for this is that to counter piracy they have some kind of protocol that the monitor might need to comply with (I say 'might' because I've seen reports saying it wasn't enabled). So, to play blu-rays I have some kind of hack that uses a beta 'makemkv' to buffer the blu-ray data and then xbmc to stream the movie (I think the instructions for this are on the 'lifehacker' website). It doesn't quite do the trick though; quite often the colour in the picture is 'reversed', i.e. it looks like a photo-negative.

Best of luck, readers!
July 12, 2012 12:00:56 AM

Awesome, thanks for the wealth of info! I've decided to go the i7 3770K and Z77 mobo route for now, as it's the best balance for what I'd like to use it for i.e. 50/50/50 photo/video editing, gaming, and virtualization experimentation. I would've thought that all CRT monitors use analog inputs, and thus would work with any DRM system; very crafty of the Blu-Ray Disc Association if they specifically built in that incompatibility, but wouldn't surprise me an iota. Cheers!
August 23, 2012 3:13:51 AM

You can't watch blu-ray in full 1080p HD unless all devices in the stream support digital encryption. It's called HDCP. From Wikpedia:
-------------------------
High-bandwidth Digital Content Protection (HDCP) system to encrypt the data sent by players to rendering devices through physical connections. This is aimed at preventing the copy of copyrighted content as it travels across cables. Through a protocol flag in the media stream called the Image Constraint Token(ICT), a Blu-ray Disc can enforce its reproduction in a lower resolution whenever a full HDCP-compliant link is not used
------------------------------

If you try to send the blu-ray playback to an analog device like an old CRT monitor, at best the resolution will be dropped down, at worst it may not play. DRM sucks.
!