Huge Workstation Build

Earlier this week I got the following components from a local computer shop:

Tyan Tempest i5400PW Motherboard
2x Xeon X5472 (3.0Ghz, 1.6Ghz FSB Harpertown)
16x 1GB Kingston DDR2-800 FB-DIMM
2x 500GB Seagate Barracuda 7200.10 Hard Drives
Silverstone Tenjim 10 (TJ10, windowed version)
Enermax Galaxy 850W
7x Noctua NF-P12-1300 120mm 9-bladed fans
2x Thermalright HR-01-X Xeon Heatsinks
FB123 Zalman Fan with Retention Bracket

Should be interesting, eh? There's no dedicated video card because it'll primarily be a HPC rig. This motherboard is absolutely great: it would support up to 64GB of memory without a problem. It's basically what the Seaburg chipset, the one in the skulltrail platform, was designed for: this really is a server, and isn't a server chip pretending to be a cool gaming-oriented platform. Having said that, it's the same chipset, so this is basically another take on the Skulltrail platform.

I unpacked everything and proceeded to the build. I have a few pictures here that I'm about to upload and will post the links very shortly. It's already up and running. I'll also install an OS and try and run some benchmarks. Here are two:



The two OEM processors and one of the Thermalright HR-01-Xs. Apparently, the X5472 is more widespread as OEM version for system integrators. The guy I bought this from said he couldn't find boxed top-range Xeons anywhere. So no stock coolers, but... who cares? Honestly, against the HR-01-X?...



The motherboard, still inside protective bag. Quite a big motherboard: I've never seen one of this size. And certainly not with 16 memory slots!

A few more...



The assembled system, closed. OK, so I still haven't bothered to hide the fan cables and still haven't zip tied the front panel cables, but come on, this already looks quite good! System is off here.





Two more from other angles. In this last one, you can see that I've done the cabling for the hard drives with zip ties in a more respectable fashion. I'll do the rest later.



This is what I ran first: a memtest86+. Doesn't recognize processor or chipset, but did you see that memory speed? (remember, the theoretical maximum bandwidth for quad-channel DDR2-800 is 25.6GB/s) I'll run something a little more beefy later on.

You might want to use a dedicated graphic card since onboard graphics uses the system RAM (even though it's only 32MB) the onboard will have a negative effect on RAM bandwidth. Just grab a cheap PCIe card like a 8400/8600.

Wow...just...wow.

I looked up those X5472 CPU's and they ring in over a grand a piece...yikes! If only I had such money!

If this rig could be used for gaming, I wonder how well it would play Crysis.

Well that is pretty sweet . Awesome in fact...
On a side note, sorry to go slightly off topic - I know Xeon's and Opteron's are designed for server platforms but if they were put with a decent graphics card how do they compare to desktop gaming platforms? With 16gb it can't be that bad at gaming right?

supremelaw, funny story about the guy who asked if a 4-socket rig could do word processing. The only sad thing is that people who don't understand this will probably frown away from the system's price without acknowledging that this is, in fact, a ridiculously powerful number cruncher that is not like those cheap desktops you pick up at your local marketplace...

dobby: it'll primarily be used for processing custom physics code for an important professor at a local university. He and a bunch of students have parallel code up and running at it will really make a difference for them!

Also, as for price, we paid in brazilian reais, and we probably paid quite a respectable amount of import taxes, so it's not easy to compare with prices you'd pay in other countries. If you think about prices in the US, you're looking at ~US$2500-2600 for mobo+CPUs, ~US$1000 of memory, US$600 case and PSU, ~US$200 for hard drives, US$140 for 7 high-end noctua fans, ~US$110 for the HR-01-Xs, and 5 or 10 dollars for the Zalman retention bracket, which, come to think of it, was really a good idea for the FB-DIMMs. This adds up to a total cost of US$4550-4650, which, actually, isn't that expensive for such a ridiculous system.

For the sake of comparison, the top alienware system (the ALX), without monitor, costs US$5149 in its default configuration. It's a X38-based desktop with a quad-core penryn extreme-edition chip overclocked to 4Ghz equipped with a petty 2GB of DDR3-1333 and 320GB (2x160GB) hard drives, but it has a dedicated video card. OK, it uses liquid cooling and clocks higher, but it doesn't feature a 1.6Ghz FSB (or two) and is no match for the throughput of this workstation for HPC. Plus, it's more expensive! But the extra US$500-600 could be put to use in a dedicated video card. But, anyway, for HPC this is appropriately priced against ultra-high-end desktops.

Out of curiosity, we could have upgraded the memory to 32GB total system memory for an extra US$1000, and if you were really hell-bent on having your own private memory nirvana, you could get 64GB for an extra ~US$5200 (ouch)! We didn't have the money to do so, but the guy from the store even gave me some prices in brazilian reais for these higher-density modules (2GB and 4GB DDR2-800 FB-DIMM): he could have obtained them if we had the need/money!!!

HP's blackbird high-end desktop also doesn't match this and costs north of US$5K. Plus, hey, I got to build the damned thing and it was fun!!!

Too bad this computer won't also be used as a development workstation - there won't even be a monitor involved. If that was the case, you could plug in, say, a rather cost-effective 8800GT or something and a 24" widescreen LCD. Now that would be a perfect place to write, debug and run code.

Over the next few weeks I will try and run some benchmarks and stability tests. I have some n-threaded code that I'm eager to test on this server. I'm also interested in comparing the performance this system achieves to a few other reference systems. What I have available consists of one Q6600-based system, one C2D E6700 and one 16-CPU numalink-connected Itanium server (CPUs @ 1.5Ghz) to test. Personally, I think the Itaniums will get trounced, with this system besting a 16-way Itanium, but I'll tell you in a few days.

Sorry for not posting any more pics. The "über-rig" is not here with me at home, it's at the university. I'll post more pictures later. Also, I need to do the final wiring. But not today. Now I'm off to bed!

I wonder what kind of number crunching you do if you do it mostly on Windows machines. From what I've seen, my guess is you're running fairly short-run (less than a day's runtime) jobs that have small (<2 GB) working sets on 32-bit Windows. I also will guess that you are running some pretty high-dollar proprietary software on some of the machines (e.g. MATLAB) and you're running them on machines that are basically desktops with all of the RAM slots filled with maybe a few dual-socket workstations thrown in there. That basically describes the only Windows number crunchers I've ever seen, and it wasn't all that many. Almost all of the number-crunchers I've come across have been big headless machines running a Unixy OS such as Linux, BSD, or Solaris and you used ssh to log into them remotely. The Windows machines ran the high-dollar proprietary software programs that worked on problems that took usually at most an hour to crunch whereas the custom-written stuff that required a lot of memory and runtime ran on the Unixy hardware. I am sure a few guys do code HPC stuff in C# but everything I've seen has been FORTRAN, C, or C++ and little of it has any sort of a GUI- the most you would do is have the number cruncher spit out a table or other results file that you'd open and look at on your desktop or laptop and that would be the GUI.

you seemed to get the wrong end og my pointy stick. all i meant is that many (NOT all) people i know who are doing reaseach in Uni can make realitively quick and effient code on the .net platform and compile it to x64 withg relative ease. however i should have said that this is really for modelling on a smallish scale. ie a few days. then the program is modfied and re run.

i did say that for serious stuff that solaris would be a 'better' choice.

also i was asking hiom a question not for a second saying i know more than him and especially not telling him what to do, for im intriguied to what he will use the system for.

Shadow: i do remember the guy who mentioned CUDA. which reminds me that the stanford university's folding at home project has a ATI client which runs on ATis high end cards.

Get the latest version of Memtest, 2.01 if they haven't updated it in a few weeks. Much more support for chipsets, however I still run into some that it doesn't recognise. That is one crazy-ass bandwidth number, crazy! I get all warm and fuzzy when I get a system to post over 5000MB/s for that number, and that is usually with the ram at way over 1000mhz and a highly overclocked CPU/FSB. If that number is accurate, all I can say is incredible.


Great pics! Thanks for posting and sharing!

The RAM looks wicked! Zalman's FB123 for the win! But is it necessary? The RAM isn't being pushed too hard, low 1.8 voltage, does it even get warm?

Thing is, the fan is blocking the view of the RAM. It can also cause detrimental turbulance, blasting down right near the exhaust. You will probably get excellent cooling via crossflow air in that system, without the FB123. There really will be quite a breeze going by there on the way out of the case, and I am betting the FB123 actually hinders it while blocking the sweet view of so many sticks of RAM!

I have a drawer full of FB123 brackets, I buy them just for the 90mm fan and screw with the big brass bolt. The little aluminum spacer cone comes in very handy too, I'll try and dig up a pic of what I've done with these.

Or do word processing!


Tips from my Anal Cable Management Guide:

Hard Drives: I may be wrong, but I think you can mount the drives facing the other way, then route the cables through to the back. Rather than running the sata lines towards the window, run the straight out the back. You probably won't even see the white zip ties any more. There should be gobs of room back there.

While you are at it, gather up all the slack in the PSU lines and stuff them back there too. One big bundle going to the back will really tidy things up.

Front Panel Wires: see if you can get your hands on about 12 inches of 1/4 inch shrink tube. Use about half of that to bundle up the FP wires from right at the mobo to about 6 inches or so -- till they are out of sight behind the hard drive bays. This will hold them at even lengths from the mobo to where they are out of sight. Use the remaining 6 inches to bridge the gap from inside the FP to inside the case. Any uneven lengths just bundle up inbetween and out of sight, behind the drives.

Fans: the fan tails are sleeved, so should easily tuck away out of sight. Like the lower CPU fan, pull all that slack in behind the drives -- rotate the fan so the where the tail comes out is closest to that gap behind the drives. You will never see a single millimeter of the cable this way. For the FB123 fan, tuck the excess in behind the PSU, or even better (if there is enough length), tuck it under the lower edge of the mobo.

If you keep the RAM fan, bolt it to the bracket on the higher side, so it is farther away from the RAM. The one pic shows it is really close, which will make much more noise. Moving it away by an inch will help a lot while not really impacting performance.


That is one slick system my friend, good job!

.

I've just installed Windows XP for a test run. OK, OK, I know, it'll only detect ~3GB of memory, but I still thought it was an OK test to begin with. I'm now running 8 instances of prime95 for stress testing and for some thermals. I'm interested in seeing just how well these HR-01-Xs perform...

Nice. Run some benches too.

Only??? Over 5k is actually an impressive number!

The only thing I have running right now that will produce near 5K is an old Prescott P4-640 in a DFI Dark P35-T2RS with 2GB G.Skill 6400's.

Using Memtest86+ 2.01 (http://memtest.org/).

Stock speed 200mhz FSB (3.2ghz), ram at DDR2-800 w/ 5-5-5-15 timings:

http://img329.imageshack.us/img329 [...] 9csbv6.jpg

Overclocked to 250mhz FSB (4.0ghz), ram at 1000mhz w/ 5-5-5-515 timings:

http://img329.imageshack.us/img329 [...] 0csnl8.jpg

At stock speed, the RAM's bandwidth is 3379 MB/s. At 4ghz/1000mhz, the RAM's bandwidth is 4230 MB/s.

The highest number I have recorded for this is 5410 MB/s with an eVGA 680i SLI (A1), e6750 Conroe, 2GB Crucial Ballistix Tracer 8500s at 520mhz FSB (3.640ghz), RAM at 1040mhz.

I've found these numbers can be eroneous when the chipset isn't recognised. Give Memtest 2.01 a try, see if you get different bandwidth numbers.

http://memtest.org/

I had no idea they get that hot!

Still, I am hopeful that crossflow air wiould be adequate. If not, it's still possible to mount a fan blowing across the ram to the exhaust, instead of blowing down on it and blocking that impressive view of so many sticks! The FB123 bracket doesn't allow for such a mount without modification or fabrication of an extra bracket. I've used Sunbeam's Anywhere Rack for these situations, and in fact made my own out of a few pieces of shevling angle iron (few bucks at any home hardware store).

Easy to test using the old finger thermometer. Cut the power to that fan, load the system with memtest or other, then check the temperature by placing the back of a finger (knuckle side) on the top of various dimms. If it burns your finger, turn the fan back on! If it is barely warm, you are good to go. Anything under burning hot should be OK, but is always subject to just how much load the system gets on a constant basis.

That's unfortunate, a case of that caliber should have tons of room. This is always how it goes with cable management! It always seems to be a situation of sacrifice -- whether increased temperatures or more time/work to accomodate things.

You never get it right first time either! It takes me three attempts most of the time (except for very simple builds), and gets more difficult with every extra part or wire you add. In fact, I got in the habit of not even screwing down the mobo with more than a few screws, because I always remove it again at least once!

No matter what, it always looks better after each attempt, and at least for me, these attempts are the most enjoyable part of assembly!

In the end, when you are satisfied, you are done! The owner is always impressed because he never saw it at first. And other's opinions are just that; opinions.

For the front panel cables, I didn't mean to actually sleeve them with nylon braid (hard to get some places), but instead to use the shrink tube that usually goes on the end of the braid to stop it from coming apart. The shrink tube should be easier to get and way cheaper. Try any electronics supplier, or even better, try some automitive or industrial suppliers who usually carry all sorts of cable protection. If you still have trouble locating shrink tube, send me a PM with your address and I will mail you 6-12 inches of the stuff.

.

^ Agreed.

Supremelaw, thanks for your suggestions. Unfortunately, there is litle that can be done in terms of improving the ventilation, but I think that, with 9 120mm fans installed, ventilation is already quite OK. 90C is hot, but it's an electrical part and there's not much else I can do to change it. The next step in improving this temperature wouldn't really be improving ventilation, it would be changing the heatsink on top of the FB-DIMM VRD to a big chunk of copper instead of aluminum.

As for your suggestions individually:

1) PSU cannot be installed with fan facing down because then the main power cables wouldn't be long enough to route behind motherboard tray and come out where needed. So I think this is the best way.

2) Case Cover stays on. I can take it off for a quick temperature check, but there's no way I'll keep the cover off all the time. I don't actually own this system, I just assembled it, and I can assure you the owner won't be pleased if he heard that he's supposed to keep the cover off all the time...

3) Using a front bay intake is not a possibility because the TJ10 has a door that covers all bays...

Something I have been meaning to confirm with Memtest, is whether the bandwidth numbers are actually a test, or simply calculating theoretical bandwidth via the FSB/Ram speeds and timings. I have suspected for a long time that these numbers are just calculations of theoretical bandwidth. In this case, it is not a benchmark at all, it isn't actually testing anything.

Nevertheless, they are still very useful in a number of ways, and notably (to quote you), to "identify if something is terribly wrong".

It is extremely consistant, which is another thing that leads me to believe it is just a calculation. In other words, at a given setting, the numbers are always the same. One thing I use it for is to confirm that I have all the settings correct when I change the clock. I'll boot to Memtest before the OS, and verify the numbers are the same as I have recorded earlier with the same setting. If the numbers match, it is about 90% certain that I have everything right. Can't tell you how many times it has saved me from booting the OS with the wrong settings.

If this is so, then changing from quad channel to dual channel may not make a lot of difference, because the FSB, memory speed and timings are still the same -- I am not sure on this because I have no idea exactly what benefits quad channel give over dual channel. Conversely, taking all the ram out except a single stick will make a big difference, because then it wouldn't calculate dual channel at all. Same goes for flipping the command rate to 1T from 2T. You get a huge boost in bandwidth because 1T theoretically halves the latency for memory operations. You don't end up with double the numbers though, because it doesn't half the number of operations, just the time the ram has to wait for the next operation. Adjusting memory timings can also have a huge effect, but often there is no change until significant adjustment is made -- possibly because one secondary timing is holding up the rest.

The numbers you are now getting are much more inline with what I would expect. I have seen Memtest mix up the cache and memory numbers before, so this all makes sense.

Finally, DDR2-1066 and a 2ghz FSB won't necessarily result in higher than 5400 MB/s bandwidth. Memtest may result in that number because that is what is calculated from all the factors. Using a benchmark within the OS is different. It is actually testing the speed, measuring the time it takes to shuffle x-amount of data around. You notice that while one benchmark will produce 8000-10,000 MB/s, another will only produce 5000-6000 MB/s. These benchmarks are measuring different functions; read, write and copy. I know of no benchmark that only produces one bandwidth score, it's always at least those three.

Therefore (and you can test for this), with a Memtest score of over 5000 MB/s you will likely get over 8000 MB/s read score in Everest Ultimate's Cachemem. You can download a trial version that should run all the basic memory benchmarks at:

http://www.lavalys.com/

I usually run the Cachemem test, then run a Report of all the benchmarks. I then save the results for future comparison. I am not sure what the trial version is lacking, it's possible you can't save the results, or run reports, or access the Cachemem test. But you should be able to run the tests individually.

Other than Everest, here's a few more good ones I use regularly. I assume you are OK with running these under XP?

Memset - not a benchmark, but displays a comprehensive list of memory timings.

MBench - I think the Memset people make this too, very quick test of memory bandwidth that outputs to a text file:

Hexus PiFast - another quick memory test with fine granularity so that you can see even minute increases in performance.

SuperPi - I run the 1M and 32M tests only, and can also help determine minute performance differences.

SiSoftware Sandra Lite XI.SP2 - Longer tests, and more of them. Plenty of reference systems built in to compare to. Save your results as you go for future reference.

CINEBENCH - Not my favorite, but I run it anyway! Fast-ish. Good results browser, save your results as database or text.

Systool - I only use this for the quick memory bench, as well as seeing and adjusting hard drive acoustic management. It will want to auto-update if connected to the internet -- I have found that it will cease to function with certain chipsets if allowed to update. If so, simply uninstall and reinstall it.

Enjoy!

.

Thanks for all that input lithotech.

A few words on your suggestions:

1) My SATA cables are too short to do what you described, and unfortunately, I don't have any longer cables, so I think I'm squarely out of options there. But the case is already much cleaner than before because all fan cables are tucked away. I forgot to post a picture.

2) The front intake is a good idea, but as I said, this can't be done with the TJ10. I think the TJ09 didn't have the front "door", but the TJ10 has it and, unless you keep the door open all the time, a bay fan is out of the picture.

As for your other ideas regarding ventilation, they're all quite good. I'll try them! In particular, I'll see if there's any air coming in from somewhere it's not supposed to.

And about benching: I have already uninstalled windows and installed ubuntu. I'll give you a benchmark post in a few minutes, hang on...

Thanks 4 the info.

Scientific linux is what you want.

https://www.scientificlinux.org/



It is based on the Red Hat Enterprise build, and will run scientific/engineering programs that Ubuntu and Fedora simply won't. Been there and got the T-shirt.


Oh, and its free too.



Forget about XP x64 - its a POS.

Some more single-threaded benchmarks this time, with one added system: an Itanium 2 @ 1.5Ghz. It's an old system, to be sure: I think the processor is about 3 years old, but we have a large (60-CPU) server here that I had the opportunity to test. Take a look at the results:





Test Description: Floating Point Special Functions is like that previous multithreaded benchmark, except it only uses 1 CPU this time. The same goes for Matrix Multiplication, and I added another test based on a high-quality PRNG (pseudo-random number generator). It's the Mersenne Twister, a generator capable of generating 10^6000 numbers before repeating itself. It's a fast-executing, brilliantly-written program that I've used extensively while doing some discrete systems research in the past. This PRNG algorithm was first published in the late 90's, I think.

Test Results: The tests largely mimic the previous ones, but the added Itanium CPU provides for some interesting results. Itanium is particularly well-suited to matrix multiplication (bear in mind this is a 3-year old CPU) and does OK even by today's standards. It's not an outlier in this chart, and you have to take into account that this is a pre-Core-2-Duo CPU. It doesn't really do well in special functions, but for some bizarre reason, it almost dies when given a PRNG to work on. It's six times slower than a single X5472 core. There probably is a reason for this (not being out-of-order?), but I have absolutely no idea of why this is so. It's interesting, nonetheless.

Our Itanium servers are basically 3 nodes consisting of 16 processors each, and I was interested in finding out how they stacked up against this new Xeon system. Apparently, one 16-CPU Itanium 2 system won't match the 8-way xeon in "special functions" but will still be somwhat faster if given a task involving larger matrices because of much larger CPU count. The Itanium 2 system will get eaten for breakfast if asked to do random numbers, though. Itanium is definitely a different architecture...

I'm about to test the Intel compilers and see if I get different results. The Intel compilers are known to be much faster than g++ in several occasions, and they're the compiler of choice for us here. I'll mess around with the compiler flags too, to see if things change.

Itanium's strength is pure 64 bit code. Ask it to do other stuff outside of that, and it suffers.

For me, its slower than K8 and Conroe based chips - but it has a far larger number of processors on the clusters than my workstations. Brute force and ignorance can work too.

Very useful updates. Thanks for taking the time to post them.

^Exactly what I was thinking. Isn't LINPACK the bench they used to determine TOP 500 supper computers?

Also if you don't mind ( i am kind of a n00b when it comes to Linux) could you point me to a HOW-TO guide for running Linpack? I am interested in running that bench on one of the servers I built a while back and see how well it dose. The person I built the server would be updating to Quad core XEONS soon.

Not harder to reach when the case is on top of a desk!

It's funny you should mention the TT Armor, too! Since it is an example I would use to describe a lower mounted optical.

My daughter's system is in an Armor Junior, which sits on top of her desk. We mounted the optical in the lowest slot at the very bottom of the case, and it is actually very very handy there! It facilitates much easier access to the drive to put in or take out disks, and frees the upper bays for the HD cage. Ever since then, I've always mounted the opticals in the lowest part of the case whenever the case allows for it!

My main system is in an Antec P180, and I just picked up one of the new Antec 300 cases for another build. I fully appreciate the lower mounted PSU design, it's only drawback is difficult cable management.

.

I've created a 4GB ramdisk under Ubuntu. It's ridiculously easy.

Some quick benchmarking shows the performance:
Hard Drive (Seagate Barracuda 7200.10, 500GB, 16MB cache):
Buffered Reads: 66.9MB/s
Ramdisk (4GB of DDR2-800)
Buffered Reads: 2728.8MB/s

Obtained using hdparm -t. Quite interesting, isn't it?... And you wouldn't even need to rewrite code to use this 40x data transfer increase... Supremelaw will go nuts over this data.

I could have gone all the way and used up, say, 12GB of memory. A Ramdisk of this size could be used to hold a lot of frequently-used programs. It would, however, still be necessary to do all appropriate copying and backups when starting the system, but once set up, it would be a very good and fast file system!