VMware ESX/Storage Server build...

[madnj]

Hi,
I'm kicking around the idea of building a server to act as a home storage server and a VMWare ESX server to provide me the capability of standing up VMs quickly and easily to be able to test/work with many different OSs and configurations. I work as a Senior level SE for an IT outsourcing group, so I'd like the ability to build/test various different configurations in a single physical system.

I'd also like to be able to use this system as my storage server, so I'm looking at RAID 5 with 4 1TB drives. I'd look to stand up a VM as a file server to which I'd allocate a lot of disk space and use that for media storage.

I'm also looking to keep the server footprint small (Medium ATX tower, not full).

The issue I am running into is that VMWare ESX server is picky about its Hardware Compatibility List, and I'm not entirely sure what challenges I might run into. I'd like to nail down my configuration before ordering.

Here's what I'm thinking:

1 x ASUS Z8NA-D6C Dual LGA 1366 Intel 5500 ATX Server Motherboard - Retail - $259.99

2 x Intel Xeon E5520 Nehalem 2.26GHz 4 x 256KB L2 Cache 8MB L3 Cache LGA 1366 80W Quad-Core Server Processor - Retail - $379.99 x 2=$759.98

3 x Kingston 4GB (2 x 2GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1066 (PC3 8500) Server Memory Model KVR1066D3S4R7SK2/4GI - Retail - $129.99 x 3=$389.97

4 x Western Digital Caviar Black WD1001FALS 1TB 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5" Hard Drive - OEM - $109.99 x 4 = $439.96

1 x Adaptec 2258100-R PCI Express SATA / SAS 5405 Kit Controller Card RAID levels 0, 1, 1E, 5, 5EE, 6, 10, 50, 60, JBOD - Retail - $379.99

1 x Athena Power BP-SAC3141B HDD internal backplane - Retail - $91.99

1 x COOLER MASTER RC-690-KKN1-GP Black SECC/ ABS ATX Mid Tower Computer Case - Retail - $79.99

Subtotal - $2401.87

I'm pretty sure everything is on the VMWare ESX server HCL (I know the Adaptec SAS Raid card is and I'm pretty sure the Intel NIC on the motherboard will be supported), but I'm not sure if I'm missing something or if there's a better choice for some of my selections. Will this work?

 

[specialk90]

Highpoint 4320 $390, 8 ports, SAS/SATA, IOP348 1.2GHz, normally $600+
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115056
This card lets you expand whereas the 4port Adaptec does not.

How many VMs will you be running at the same time? Will the VMs be running while your VM-File Server is being used?
I ask this because your current storage setup might be a bottleneck. Now that I think about it, you certainly want a separate drive for the OS, preferably 2 for Raid 1.
- What host OS, and what guest OS's and File Server OS?
- Did you buy ESX already or have you thought about ESXi ?

If you run multiple VMs and the host OS on 1 Raid 5 array, the performance will plummet. Because you chose a dual-CPU setup, I assume you will be running several VMs at once, which means a single Raid 5 array will not be enough. If you don't need to run several VMs at once, then an i7 system would work just fine since the i7 has Hyper-Threading also.

 

[madnj]

I'll check on that card, but I'm not sure it's supported on the VMWare ESX Hardware Compatibility List.

I'll probably be running 10 or so VMs at once when I get things really cranking, but initially probably just 3-4.

I understand the concern with Host OS being a storage bottleneck, but I'm actually going to be running full ESX which doesn't require a "host" OS. The entire VMWare system runs on a Hypervisor (basically a really slim 'nix based kernel which interfaces between the vmware systems and the physical hardware) and the entire ESX VM infrastructure is managed via a management console that sits on another PC. That's the beauty of ESX honestly, it's SO slim and it has no overhead since the OS's are truly running in parallel and not fighting a host OS for resources. I will definitely be running my File server all the time, but it won't be using CPU clock cycles when it's sitting idle (another benefit of ESX).

Also, ESXi is basically ESX lite with most of the core functionality available minus things like Vmotion and some of the other Enterprise stuff. I may be able to get myself a working VMWare license from work though because we are partnered with VMWare and one of my coworkers has a very good relationship with them.

I'll likely be running Server 2003, Server 2008, Windows Vista/XP/7, Linux, and from an application standpoint I'll run Exchange 2007, System Center Config Manager (SCCM), SQL, possibly Sharepoint, and some of the CA IT Client Manager tools that I manage for our outsourcing group.

I've seen 50 VMs running on a 4 node VMware cluster with fewer cores per node, but running with EMC SAN storage connected via Fiberchannel and it absolutely screamed. While I don't expect quite that much out of the disk array, I expect by using a full hardware SAS RAID controller, I'll get some pretty solid throughput.

I may look to get a SAS RAID card with more ports, but I'm not sure it's necessary at the moment.

I do appreciate the comments though, and am continuing to weigh in my head things I might want to add to this build. I've already identified that I'll need CPU coolers because the retail Xeons don't come with any. I'm also mulling over going with a full tower case and a more feature rich server motherboard with Lights Out management, but I just don't know that I need it.

Thanks again, and don't be shy folks, I'm looking for any comments or insight that any of you can provide.

 

[boonality]

I "think" that ESX server requires iSCSI... that's a big expense and you may want to look into whether or not it is required.

 

[specialk90]

I don't see a Power Supply or Video card in your list. I recommend PC Power & Cooling, the 610 or 750. For $5 or $6, they make a Molex to 8pin CPU power adapter, so no need for a full-fledged server psu.

Since you will be using VMs, using Raid 10 should provide you with much better performance. Have you considered 8 drives and run 2 Raid 10 arrays?

I totally had a brain fart with the ESX & host OS. I'm used to people asking about Server or Workstation(and I use Workstation).

 

[madnj]

I "think" that ESX server requires iSCSI... that's a big expense and you may want to look into whether or not it is required.

Nah, ESX doesn't require iSCSI, it does require that the storage for the VMWare hosts be presented via a SCSI interface though (there's no native support for SATA drives or drive arrays). What's nice about SAS controllers (Serial Attached SCSI) is that you can connect SATA drives natively and they are presented to the OS as pure SCSI devices.

The biggest issue though is that when installing ESX your disk controller and network adapter have to be natively supported by the ESX installation or it won't install, and there's a limited list of supported hardware on the ESX Hardware Compatibility List. The Adaptec Card I selected is on the list, so I should be good there.

 

[madnj]

I don't see a Power Supply or Video card in your list. I recommend PC Power & Cooling, the 610 or 750. For $5 or $6, they make a Molex to 8pin CPU power adapter, so no need for a full-fledged server psu.

Since you will be using VMs, using Raid 10 should provide you with much better performance. Have you considered 8 drives and run 2 Raid 10 arrays?

I totally had a brain fart with the ESX & host OS. I'm used to people asking about Server or Workstation(and I use Workstation).

I currently have an Antec 750 watt power supply which I was planning on using for this build, but I totally forgot that I'd need the 8pin CPU power adapter. I'll definitely add that to my order (if I don't order a new PSU). I haven't considered 8 drives because I don't need that amount of storage, and I was looking to keep the footprint/power requirements down. Also, with 8 drives there'll probably be more maintenance needed for blown drives.

I actually might order a 5th drive to have on standby for quick replacement in case of a drive being marked as bad by my disk array, but I'm still looking to keep this server to a medium tower case.